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Abstract 

NASA Langley Research Center has developed a library that allows Intel NX mes- 
sage passing codes to be executed under the more popular and widely supported Parallel 
Virtual Machine (PVM) message passing library. PVM was developed at Oak Ridge Na- 
tional Labs and has become the defacto standard for message passing. This library will 
allow the many programs that were developed on the Intel iPSC/860 or Intel Paragon 
in a Single Program Multiple Data ( SPMD ) design to be ported to the numerous ar- 
chitectures that PVM ( version 3.2) supports. Also, the library adds global operations 
capability to PVM. A familiarity with Intel NX and PVM message passing is assumed. 


1. Introduction 


At NASA Langley Research Center (LaRC), the center’s vector supercomputers have 
become heavily saturated with users’ jobs. Alternatives are being considered to off load 
some of these jobs to other systems. Among the alternatives considered is the transition 
of some applications from the vector supercomputers to parallel machines and workstations 
clusters. With the proliferation of high powered workstations, workstation clustering, in 
both batch and parallel use, offers an attractive solution to supercomputer saturation. 

At NASA LaRC, the Parallel Virtual Machine (PVM) software provides the most popular 
parallel programming environment. PVM was developed at Oak Ridge National Laboratory 
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and has become a defacto standard for message passing (ref. 4) . But before PVM had 
reached this level of popularity, many parallel applications had been developed on the Intel 
iPSC/860. There was a need to transition these Intel NX message passing (ref. 3) codes to 
PVM. 

This document describes the Intel to PVM, version 3.2 (PVM3) libraries. A familiar- 
ity with Intel NX and PVM message passing is assumed. The libraries, Iibi2pvm3.a and 
Iibfi2pvm3.a, are written in C and contain wrappers for several Intel functions and routines. 
The executable, pvmexec, is a C program which starts the PVM daemons, runs the user 
application, waits for completion of the slaves, and terminates the PVM daemon processes. 
If pvmexec is run in the Distributed Queing System (DQS) environment (ref. 5), then the 
PVM daemons will not be started or stopped by pvmexec. pvmexec is able to detect if it is 
being run in DQS and will relinquish PVM daemon control to DQS. 

The main purpose of the libraries is to allow the user with a code written for a Intel 
Message Passing Supercomputer in C or FORTRAN to quickly port the code to a worksta- 
tion cluster using PVM3. To use the libraries in conjunction with the executable pvmexec 
( pvmexec is analogous to cubeexec 1 ), the user must add two subroutine calls and convert 
asynchronous message passing calls (e.g., isend and irecv) to synchronous calls (e.g., csend 
and crecv). 

Another use of the libraries is to give the PVM3 user access to many of the global libraries 
which are absent in the standard PVM distribution. To use the global routines without using 
pvmexec, the user should make a call to the pvmsetup routine (see section 4). After the 
task ids and number of slave processes are known, the pvmsetup routine is called so that 
the global routines can be used. 


2. Building the libraries 


This library is available via anonymous ftp from 


blearg. larc.nasa.gov: /pub /pvm/i2pvm3 .shar .Z 


Before unpacking i2pvm3.shar , the user should make the directory $H0ME/pvm3. To unpack 
the library, the following should be typed in the user’s home directory: 


% sh i2pvm3.shar 


1 cubeexec was developed by William J. Nitzberg from the Numerical Aerodynamic Simulation (NAS) 
Systems Division at NASA Ames Research Center to easily run executable code on the iPSC/860. cubeexec 
is not an Intel supported utility. 
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The following should be typed to compile the library: 


% cd pvm3/i2pvm3 
% make 

This will compile the libraries and pvmexec.c. The libraries are moved to 
$PVM-ROOT /lib /$PVM. ARCH. The executable, pvmexec , is moved to 
$PVM.ROOT/bin/$PVM.ARCH. The include files that the user will need are installed in 
SPVM.ROOT /include. 

To compile a program to run in the PVM environment, the following libraries should be 
linked in this order: (UbfiSpvmS.a), Iibi2pvm3.a and libpvmS.a. The following is an example 
compile line for a C and FORTRAN program, respectively: 

% cc -0 -o daria daria.c -L$PVM-ROOT/lib/$PVM-ARCH -U2pvm3 -lpvm3 
% f77 -0 -o daria daria.f -L$PVM.ROOT/lib/$PVM.ARCH -Ifi2pvm3 -Ii2pvm3 -IpvmS 


3. Use of pvmexec 


The libraries can be easily used in conjunction with the executable pvmexec. pvmexec 
starts up the daemon processes, runs the application, waits for the application to finish 
then kills the PVM daemons. If pvmexec is run in the Distributed Queing System (DQS) 
environment (ref. 5), then the PVM daemons will not be started or stopped by pvmexec. 
pvmexec is able to detect if it is being run in DQS and will relinquish PVM daemon control 
to DQS. 

When using the library with pvmexec , the first executable line in the code should be a 
call to pvminitQ. This routine receives messages from pvmexec. The final call in the user’s 
program should be to pvmquitQ. Failure to call these routines by ALL processes 
will cause pvmexec to hang. Once the pvmquit() routine has been called by all processes, 
pvmexec will kill the PVM daemons and exit. As noted before, the user must also convert 
asynchronous routines to synchronous routines. 

pvmexec recognizes the three options -t, -v, and -V. Option -t is used to specify the 
number of processes to start, -v is verbose mode, and -V prints the version of pvmexec. An 
example for running four processes of the executable node : 

% pvmexec -v -t 4 node 

pvmexec will start daemons on all of the hosts in hostfile. hostfile is a PVM host file (ref. 1) 
and is read from the directory in which pvmexec is executed. If hostfile is not present, 
pvmexec will run the all PVM processes on the current workstation. 


3 



4. Use of libraries without pvmexec 


The libraries can be used without using pvmexec, however, it is the user’s responsibility 
to start and stop the PVM daemons (see (ref. 1) for more information). To use the libraries 
without pvmexec , make a call to pvmsetup after the task ids and number of slave processes 
are known. NOTE: if using pvmsetup, do NOT call pvminit or pvmquit. The following 
code fragment is an example on how to use pvmsetup. 

C example: 


mytid = pvm_mytid() ; 
tids[0] = pvm_parent() ; 
if( tids[0] < 0){ 
tids[0] = mytid; 

pvm_spawn("spmd" , (char**)0, 0, NPR0C-1, &tids[l]); 
pvm_initsend( PvmDataDef ault ); 
pvm_pkint (tids, NPROC, 1); 
pvm_mcast (frtids [l] , NPR0C-1, 0); 

> 

else { 

pvm_recv(tids [0] , 0); 
pvm_upkint (tids , NPROC, 1); 

> 

pvmsetup (tids, NPROC) ; 


FORTRAN example: 

call pvmfmytid( mytid ) 
call pvmf parent ( tids(0) ) 
if ( tids(0) .It. 0) then 
tids(0) = mytid 

call pvmfspawn( 'spmd', PVMDEFAULT, , NPR0C-1, tids(l), numt ) 
call pvmf pack ( INTEGER4, tids, NPROC, 1, info ) 
call pvmfmcast( NPR0C-1, tids(l), 0, info ) 
else 

call pvmfrecv( tids(0), 0, info ) 
call pvmf unpack ( INTEGER4, tids, NPROC, 1, info ) 
end if 

call pvmsetup ( tids, NPROC ) 
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5. Supported routines 


Routine 

Usage 

Description 

Sending 



csendQ 

csendsi() 

csendi() 

csendr() 

csendd() 

csend(msgtype, buf, len, node, pid) ; 
csendsi(msgtype, buf, len, node, pid); 
csendi(msgtype, buf, len, node, pid); 
csendr(msgtype, buf, len, node, pid); 
csendd(msgtype, buf, len, node, pid); 

send a message 

send short integer message 

send an integer message 

send a real message 

send a double precision message 

Receiving 




crecv(msgtype, buf, len); 
crecvsi(msgtype, buf, len); 
crecvi(msgtype, buf, len); 
crecvr(msgtype, buf, len); 
crecvd(msgtype, buf, len); 

receive a message 

receive short integer message 

receive an integer message 

receive a real message 

receive a double precision message 

Global 



gdhigh() 

gdlow() 

gdprodQ 

gdsum() 

gihigh() 

gilow() 

giprod() 

gisum() 

gshigh() 

gslow() 

gsprod() 

gssum() 

gsync() 

gdhigh(buf,num,work); 

gdlow(buf,num,work); 

gdprod(buf,num,work); 

gdsum(buf,num,work); 

gihigh(buf,num,work); 

gilow(buf,num,work); 

giprod(buf,num,work); 

gisum(buf,num,work); 

gshigh(buf,num,work); 

gslow(buf,num,work); 

gsprod(buf,num,work); 

gssum(buf,num,work); 

gsyncQ; 

global double precision MAX 

global double precision MIN 

global double precision MULTIPLY 

global double precision SUM 

global integer MAX 

global integer MIN 

global integer MULTIPLY 

global integer SUM 

global real MAX 

global real MIN 

global real MULTIPLY 

global real SUM 

synchronization 

Other 



pvminit() 

pvmsetup() 

pvmquit() 

mynodeQ 

numnodes() 

cprobe() 

infocount() 

infonode() 

dclock() 

pvminit(); 

pvmsetup(tids,nproc) ; 
pvmquit(); 
int mynodeQ; 
int numnodesQ; 
cprobe(msgtype) ; 
int infocountQ; 
int infonodeQ; 
double dclockQ; 

call when using pvmexec 
call when NOT using pvmexec 
send quit signal to pvmexec 
returns logical process number 
returns number of processes 
wait for a message to arrive 
length of message in bytes 
node id for sending process 
returns wall clock in seconds 


Table 1: Supported C routines 











Routine 

Usage 

Description 

Sending 



csend() 

csendsiQ 

csendiQ 

csendrQ 

csendd() 

call csend(msgtype, buf, len, node, pid) 
call csendsi(msgtype, buf, len, node, pid) 
call csendi(msgtype, buf, len, node, pid) 
call csendr(msgtype, buf, len, node, pid) 
call csendd(msgtype, buf, len, node, pid) 

send a message 

send short integer message 

send an integer message 

send a real message 

send a double precision message 

Receiving 



crecv() 

crecvsi() 

crecvi() 

crecvrQ 

crecvdQ 

call crecv(msgtype, buf, len) 
call crecvsi(msgtype, buf, len) 
call crecvi(msgtype, buf, len) 
call crecvr(msgtype, buf, len) 
call crecvd(msgtype, buf, len) 

receive a message 

receive short integer message 

receive an integer message 

receive a real message 

receive a double precision message 

Global 



gdhigh() 

gdlow() 

gdprod() 

gdsum() 

gihigh() 

gilow() 

giprod() 

gisum() 

gshigh() 

gslowQ 

gsprod() 

gssum() 

gsync() 

call gdhigh( buf, num, work) 
call gdlow(buf, num, work) 
call gdprod(buf, num, work) 
call gdsum(buf, num, work) 
call gihigh(buf,num,work) 
call gilow(buf, num, work) 
call giprod(buf, num, work) 
call gisum(buf, num, work) 
call gshigh(buf, num, work) 
call gslow(buf, num, work) 
call gsprod(buf, num, work) 
call gssum(buf, num, work) 
call gsync() 

global double precision MAX 

global double precision MIN 

global double precision MULTIPLY 

global double precision SUM 

global integer MAX 

global integer MIN 

global integer MULTIPLY 

global integer SUM 

globed real MAX 

global real MIN 

global real MULTIPLY 

global real SUM 

synchronization 

Other 



pvminit() 

pvmsetup() 

pvmquit() 

mynode() 

numnodes() 

cprobe() 

infocount() 

infonode() 

dclock() 

call pvminit() 
call pvmsetup(tids,nproc) 
call pvmquit() 
integer mynode() 
integer numnodes() 
call cprobe(msgtype) 
integer infocount() 
integer infonode() 
double precision dclockQ 

call when using pvmexec 
call when NOT using pvmexec 
send quit signal to pvmexec 
returns logical process number 
returns number of processes 
wait for a message to arrive 
length of message in bytes 
node id for sending process 
returns wall clock in seconds 


Table 2: Supported FORTRAN routines 


If the PVM environment has machines with different byte ordering conventions, some 
additional code changes will be needed. This is because message passing on the Intel is 
based on sending messages in bytes. If the PVM environment has machines with different 
byte ordering conventions, the user will need to use a different set of communication routines. 


6 












These routines help PVM determine how to send the message. To use these calls, replace 
csend with csendx. where x is either si, i, r or d which stands for short integer, integer, real 
or double precision, respectively. For example, to send the real variable y to logical node 2, 
use this syntax: csendr(msgtype, y, 4 > 0). Note that the message length is still in bytes, 

so the user only needs to add the appropriate appendix to csend. This message should be 
received by using the corresponding receive routine: crecvr(msgtype, y, 4)- 


6. Unsupported routines 


Many NX routines are absent from this library. The supported routines were chosen 
based on experience in porting from the Intel/i860 to the PVM environment. Many of the 
asynchronous routines are not supported because it is difficult to emulate these routines in 
PVM. The easiest solution to this problem is to have the user change asynchronous routines 
(e.g., isend, irecv) to synchronous communication (e.g., csend, crecv). 


7. C Example 


Given the following Intel C program : 


#include <stdio.h> 

•include <cube.h> 

mainQ 

{ 

int iam, nproc; 
float x; 

iam = mynodeO; 
nproc = numnodesO; 
if ( ! iam ) { 
x = 20.0; 

csend(100, x, 4, -1, 0); 

> 

else { 

crecv(l00, x, 4) ; 

> 

gssum(x , 1 , work) ; 

if ( ! iam ) printf ("check: x should equal %d\n",nproc*20.0) ; 
printf("iam= '/,d, x= '/,f\n" ,iam,x) ; 

> 
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To run this program in a PVM environment using the Iibi2pvm3.a library, the following code 
changes would need to be made: 

1) change the include file “cube.h” to “nx2pvm.h” 

2) change the first executable line to “pvminitQ;” 

3) change the last executable line to “pvmquitQ;” 


Below is the modified C code: 

#include <stdio.h> 

♦include <nx2pvm.h> 

mainO 

int iam, nproc; 
float x, work; 

pvminit () ; 
iam = mynodeO ; 
nproc = numnodesO; 
if ( ! iam ) { 
x = 20.0; 

csendClOO, x, 4, -1, 0); 

> 

else { 

crecv(100, x, 4); 

> 

gssum(x, 1 , work) ; 

if ( !iam ) printf ("check: x should equal '/,d\n" ,nproc*20 .0) ; 
printf ("iam* % d, x= V.f \n" , iam ,x) ; 
pvmquit () ; 

> 
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The following is a makefile for compiling the program to run on a PVM environment: 


# 

INCLUDEDIR = 

PVMLIB 

BDIR 

XDIR 

CLIBS 


$(PVM_R00T) /include 
$ (PVM.ROOT) /lib/$ (PVM.ARCH) 
$ (PVM_R00T) /bin 
$ (BDIR) / $ (PVM_ARCH) 

-Ii2pvm3 -lpvm3 


CFLAGS = -g 


beavis : 

cc $ (CFLAGS) -1$ (INCLUDEDIR) -L$(PVMLIB) -o $0 beavis. c $(CLIBS) 
mv beavis $(XDIR) 


8. FORTRAN Example 


Given the following Intel FORTRAN program : 

program beavis 
include 'fcube.h' 

iam = mynodeO 
nproc = numnodes() 
if(iam .eq. 0) then 
x = 20.0 

call csend(100, x, 4, -1, 0) 
else 

call crecv(100, x, 4) 
endif 

call gssum(x, 1 .work) 

if (iam .eq. 0) write(*,*) 'check: x should equal ',nproc*20.0 
write(*,*) 'iam = '.iam,', x= ’ ,x 

end 
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To run this program in a PVM environment using the Iibfi2pvm3.a library, the following 
code changes would need to be made: 


1) change the include file “fcube.h” to “fnx2pvm.h” 

2) change the first executable line to “call pvminit()” 

3) change the last executable line to “call pvmquitQ” 


Below is the modified FORTRAN code: 


program beavis 

include 'fnx2pvm.h' 

call pvminitO 
iam = mynodeO 
nproc = numnodesO 
if (iam .eq. 0) then 
x = 20.0 

call csend(l00, x, 4, -1, 0) 
else 

call crecv(100, x, 4) 
end if 

call gssum(x, 1 .work) 

if (iam .eq. 0) write(*,*) 'check: x should equal ',nproc*20.0 
write(*,*) 'iam = '.iam,', x= ' ,x 

call pvmquitO 
end 

The following is a makefile for compiling the program to run on a PVM environment: 


# 


PVMLIB = 

$ (PVM.ROOT) /lib/$ (PVM.ARCH) 

BDIR 

$ (PVM_R00T) /bin 

XDIR 

$ (BDIR) / $ (PVM_ARCH) 

FLIBS = 

-Ifi2pvm3 -Ii2pvm3 -lpvm3 

beavis : 



cp $(PVM_R00T)/include/fnx2pvm.h . 

f77 $(FFLAGS) -L$(PVMLIB) -o $<D beavis. f $(FLIBS) 

mv beavis $(XDIR) 
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9. Executing the examples 


The program is compiled and linked by typing make. For compatibility with PVM, the 
executable beavis is moved to $PVM.ROOT/bin/$PVM-ARCH. To execute beavis over four 
machines, the file hostfile should be created with each machine name on a separate line 
(see (ref. 1) for details on how to set up a host file). To execute the code, the following 
should be typed: 

% cd $PVM-ROOT/bin/$PVM-ARCH 
% pvmexec -v -t 4 beavis 


Analogous to PVM, all output to the screen is redirected to the file /tmp/pvml.< uid >. To 
obtain the status of the job while it is running, in another window on any of the machines 
running PVM, the following should be typed: 


% pvm 
pvm> ps -a 


10. Summary 


This report describes the use of the NASA Langley Research Center library for conversion 
of Intel NX message passing codes to PVM3.2 message passing codes. If an application is a 
candidate for conversion, it must be of SPMD design and any asynchronous sends and receives 
must be converted to synchronous sends and receives. If the intended PVM environment is 
heterogeneous, some additional code modifications may be necessary. 

This library should enable users to quickly port codes developed on the Intel iPSC/860 
or Intel Paragon to other environments. This includes workstations clusters or even other 
parallel computers that provide PVM support. The use of pvmexec emulates the Intel NX 
environment and should minimize porting difficulties. The use of this library also adds global 
operations capability to PVM. Additions, modifications, or suggestions are welcome and can 
be sent to the authors. 
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